User interface systems and methods for viewing and manipulating digital documents

ABSTRACT

Systems, including handheld computing devices that include system code stored within the memory and adapted to be executed by the processor. The system code can process an input byte stream that is representative of contents to be displayed on the touch sensitive display and can generate a content document file representative of an internal representation of the content. A tool document file may also be stored in the memory and may provide an internal representation of a document providing an image that is representative of a graphical tool. Associated with a tool document can be tool code that is capable of processing the content document file to create an internal representation of the contents that presents the content in a manner that achieves a display effect associated with the tool. The device will also include parsing code that processes the content document file, the tool document file, and the processed internal representation to generate a screen document for display on the touch sensitive display in a manner that portrays the display effect.

REFERENCE TO RELATED APPLICATIONS

[0001] The systems and methods described herein relate to earlier filedBritish patent application No. 0009129.8 and earlier filed U.S. patentapplication Ser. No. 09/703,502, as well as the U.S. patent applicationentitled, Systems and Methods for Digital Document Processing, filedeven date herewith, all of which name Majid Anwar as an inventor, andthe contents of which are herein incorporated by reference.

FIELD OF THE INVENTION

[0002] The systems and methods described herein relate to systems andmethods for viewing and manipulating a display of digital documents, andmore particularly to user interface systems and methods that allow auser to manipulate and view a digital document presented on a display,such as the display of a hand held electronic device, such as acomputer, mobile communicator or phone; or a display device associatedwith a tactile commander.

BACKGROUND OF THE INVENTION

[0003] Today, there is a strong effort to build mobile and handheldcomputing devices that easily allow users to view documents, email,video presentations, and other forms of content. To achieve thisconvergence, engineers and scientists have developed systems includingthe systems described in the above-referenced U.S. patent applicationentitled, Systems and Methods for Processing Digital Documents, thecontents of which are hereby incorporated by reference. As describedtherein, digital content, whether a document, audio visual presentation,or some other type of content, is processed by a software systemoperating on a handheld device, mobile device, or some other platform,and translated into a uniform internal representation that can beprocessed and manipulated by the software system so that a display ofdifferent types of content can be generated by the system and presentedon the screen display of the respective device.

[0004] These systems, as well as other handheld and mobile computingsystems such as the Palm Pilot, Compaq Ipaq, and mobile phones, aretherefore capable of providing a display of content to a user. However,these handheld and mobile systems are typically limited to simple inputdevices such as small and limited keyboards, commonly the keyboardspresent on a cellular phone, or by small touch screen systems, such asthe touch screens provided with the Palm computing device. Therefore,although these systems are capable of presenting content to a userwherein the content can be quite complex, these systems have limitedcapacity to allow a user to manipulate the display of that content, suchas by paging through the different pages of a document or selectingdifferent portions of a document. Therefore, although these handheld andportable systems may be quite useful, there are limitations to their usebased in part on the user interfaces available for supporting themanipulation and viewing of the content presented on these devices.

[0005] Consequently, there is a need in the art for systems and methodsthat provide improved user interface tools that make it more facile tomanipulate and view content presented by a handheld or portable device.

[0006] Additionally, there is a need in the art for user interface toolsthat allow for the manipulation of content, when that content isseparated from its native application program.

SUMMARY OF THE INVENTION

[0007] The systems and methods described herein provide advanced userinterface tools that allow a user to more easily manipulate and viewcontent presented on a mobile device or handheld device. In oneparticular embodiment, the systems and methods described herein providea graphical user interface that exhibits a touch and feel user interfaceexperience. More specifically, the systems and methods described hereininclude hand-held or mobile computer devices having a system forsimulating tactile control over a document which may be viewed on thedevice itself or through a remote instruction or remote instruction orremote display on another unit. These systems may include a housingwhich supports a processor, memory, and a touch-sensitive display (ordisplay with remote touch-sensitive control), system code stored withinthe memory and adapted to be executed by the processor. The system codemay generate or provide a digital representation of a document, whereinthe digital representation may include data content and a page structurerepresentative of the page layout of the document. Thus, in certainapplications the rendered image can include the content of the documentas well as the layout of the document, thereby providing an image ofwhat the document physically looks like. The system may also include arendering engine that may include a parser and a renderer for renderingat least a portion of the page layout of the digital representation onthe touch-sensitive display. A screen monitor can monitor thetouch-sensitive screen for detecting movement across a surface of thetouch sensitive screen, and an interface process can process thedetected movement to detect a motion representative of a command toalter the page structure of the digital representation. A navigationmodule may be responsive to the interface process and can change therendered portion of the page layout. Thus, by altering the renderedportion of the page layout, the system allows a user to navigate throughthe digital representation of the document. Although the systems andmethods of the invention will have applicability and value when used inother applications and on other types of systems, for purposes ofillustration, the invention will be described with reference to thoseapplications where the systems facilitate the navigation of documentspresented on a hand-held computing device.

[0008] More particularly, the systems and methods described hereinprovide, among other things, computer devices having a system forsimulating tactile control over a document. In one embodiment, thesesystems comprise a processor, memory, and a display; system code storedwithin the memory and adapted to be executed by the processor, thesystem code providing a digital representation of a document includingdata content and a page structure representative of a page layout of thedocument; a rendering engine for rendering at least a portion of thepage layout of the digital representation on the display; a screenmonitor for monitoring the screen to detect movement of an object acrossan image presented on the display; an interface process for processingthe detected movement to detect a motion representative of a command toalter the rendered page structure of the digital representation, and anavigation module responsive to the interface process for changing therendered portion of the page layout, wherein altering the renderedportion of the page layout allows a user to navigate through the digitalrepresentation of the document.

[0009] These computer devices can include touch-sensitive displays wherethe screen monitor monitors a touch-sensitive screen for detectingmovement across a surface of the touch sensitive display, as well ascomputer displays capable of depicting a cursor moving across a screenof the display, and wherein the screen monitor detects movement of thecursor across a surface of the display. The processor, memory, screenmonitor and a display may be arranged as a data processing platformuseful with a plurality of applications and devices including hand-heldcomputers, telephones, mobile data terminal, a set top box, an embeddedprocessor, a notebook computer, a computer workstation, a printer, acopier and a facsimile machine.

[0010] In certain optional embodiments, the computer device may alsoinclude a velocity detector for determining a velocity vector associatedwith motion detected across the surface of the touch-sensitive display,as well as means for applying a velocity characteristic to a documentwithin a display.

[0011] Additionally, these computer devices can have an interfaceprocesses that make it more easy to navigate through a document or acollection of documents and other content. These interface processes caninclude a page-flip detector for detecting a motion across the surfaceof the touch-screen at a location presenting a portion of the pagelayout graphically representative of a corner of a document. Thepage-flip detector can render a portion of the page layoutrepresentative of a page adjacent a currently rendered page. Similarly,the device can include a page curl detector for rendering a portion ofthe page layout representative of a portion of a page adjacent acurrently rendered page. Additionally, the interface process can includea gesturing process for detecting a predefined movement representativeof a command for selecting a portion of the page layout to be rendered,or for altering data content of the digital representation of thedocument. Still further interface controls include processes forcontrolling a transparency characteristic of a document presented on thedisplay and for controlling a transparency characteristic of selectedportions of the document for adjusting visibility of the selectedportions relative to other portions of the document. Other interfaceprocesses can provide tools, including tools representative of amagnifying tool, a ruler, a text entry cursor, a thumbnail navigationcolumn, a thumbnail view of linked content and a query tool.

[0012] In still other aspects, the invention provides computer devices,and related processes, having a context sensitive graphical interfacetool. These devices may comprise a processor, memory, and atouch-sensitive display; a content document file stored in the memoryand being representative of an internal representation of the content; atool document file stored in the memory and providing an internalrepresentation of a document providing an image that is representativeof the graphical interface tool; tool code capable of running of theprocessor and being associated with the tool document file and capableof processing the content document file to create an internalrepresentation of the content that when rendered presents the content ina manner that achieves a display effect associated with the tool;parsing code that processes the content document file, the tool documentfile, and the processed internal representation to generate a screendocument for display, and; interface code capable of running on theprocessor for allowing a user to arrange the image of the graphicalinterface tool into a selected contextual relationship over the renderedcontent and for directing the tool code to process a portion of thecontent document file associated with the selected position.

[0013] The contextual relationship between the graphical interface tooland the rendered content can vary depending upon the application, andmay for example be selected from the group consisting of the relativeposition of the graphical interface tool and the rendered content, thetime at which the graphical interface tool acts on the rendered content,and the state of the rendered content. These devices are flexible andmay be embodied in different forms and devices, including, but not beinglimited to, a hand-held computer, a telephone, mobile data terminals, aset top box, an embedded processor, a notebook computer, a computerworkstation, a printer, a copier, and a facsimile machines, as well asin car systems, and domestic devices such as audio players, microwaves,refrigerators, and washing machines.

[0014] However, it will be understood by those of ordinary skill in theart that these interface tools may be employed in other applicationsincluding applications wherein content is displayed on a conventionalcomputer workstation that includes typical input tools such as astandard keyboard and a mouse. Additionally, it will be understood thatthe systems and methods described herein also provide useful tools forproviding interfaces for embedded display systems, such as embeddedvisual displays employed as output devices. Examples of such embeddeddisplay systems can include cellular phones, copy machines that includea visual touch screen display that allows a user to select differentoptions for performing a copy job and which may also present images tothe user of the documents being copied. Other examples may include faxmachines wherein visual displays are provided to a user to allow a userto view a depiction of an incoming fax. Other embodiments andapplications of the user interface systems and methods described hereinwill be apparent to those of ordinary skill in the art.

[0015] More particularly, the systems and methods described hereinprovide user interface tools that allow a user to manipulate contentdisplayed on a screen. In particular, the systems and methods describedherein provide software systems that create an abstraction layer forinformation that is to be presented on a display. This abstraction layerincludes a document object wherein a document object containsinformation, or content, that is to be displayed on a screen. In oneimplementation, all information displayed on a screen is treated as onedocument. Thus, at the highest level, the entire contents of a screen isunderstood as one document object. Further to this embodiment, it willbe understood that a document object may contain other document objects,each of which may contain a subset of the content displayed to the user.Thus, at the screen level, all information displayed will be understoodas a single document wherein items, such as web pages, streamed video,and graphical icons, presented on the screen are each understoodseparately as document objects contained within the high level screendocument object. Therefore, all content displayed on a screen is treatedabstractly as a document, and this paradigm holds whether the contentbeing displayed is information representative of a page of text orinformation representative of a user interface tool or window/desktopfurniture. Accordingly, the user interface systems and methods describedherein provide user interface tools and functionality for allowing auser to manipulate document objects presented on a screen display.

[0016] Additionally, the systems and methods described herein provide,in one embodiment, a handheld computing device that comprises a housingwhich supports a processor, memory, and a touch sensitive display.Further, the computing device may include system code stored within thememory and adapted to be executed by the processor. The system code canprocess an input byte stream that is representative of content to bedisplayed on the touch sensitive display and can generate a contentdocument file representative of an internal representation of thecontent. A tool document file may also be stored in the memory and mayprovide an internal representation of a document providing an image thatis representative of a graphical tool. Associated with a tool documentcan be tool code that is capable of processing the content document fileto create an internal representation of the content that presents thecontent in a manner that achieves a display effect associated with thetool. The device may also include parsing code that processes thecontent document file, the tool document file, and the processedinternal representation to generate a screen document for display on thetouch sensitive display in a manner that portrays the display effect.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The foregoing and other objects and advantages of the inventionwill be appreciated more fully from the following further descriptionthereof, with reference to the accompanying drawings wherein;

[0018]FIG. 1 provides a functional block diagram of one system accordingto the invention;

[0019]FIG. 2 depicts one example of a tool generated by a system such asthe system depicted in FIG. 1;

[0020]FIG. 3 depicts a graphical user interface tool that presents aplurality of thumbnail sketches for navigating through a document havinga plurality of pages;

[0021]FIG. 4 depicts a magnifying graphical user interface toolaccording to the invention which provides additional information withina magnified area;

[0022]FIG. 5 depicts a semi-transparent and adaptively sizeable rulergraphical tool;

[0023]FIG. 6 depicts a transparent query marker graphical user interfacetool;

[0024]FIG. 7 depicts a user interface mechanism for activating anddeactivating a graphical tool;

[0025]FIGS. 8a and 8 b depict a user interface tool for visuallyenhancing selected portions of a displayed document;

[0026]FIG. 9 depicts a further user interface tool according to theinvention;

[0027]FIGS. 10 and 11 depict a text entry tool according to theinvention;

[0028]FIGS. 12a-12 g depict a set of strokes for providing commands to ahand-held system; and

[0029]FIGS. 13A-13B depict a user interface tool for scrolling through adocument by applying a velocity characteristic to the document beingdisplayed.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

[0030] The systems and methods described herein include systems andmethods for manipulating and viewing documents displayed on a viewingsurface such as a computer terminal, a display screen, a printer,plotter, or any other output device suitable for creating a visualrepresentation of human readable information. For the purpose ofillustration, the systems and methods will be described with referenceto certain exemplary embodiments, including hand held computer systemsthat include touch screen displays and which are capable of displayingan integrated view of content produced in different formats. Inparticular, the systems and methods described herein include graphicaluser interface tools that are capable of presenting tools that can bepresented as content that will be integrated along with other contentbeing displayed on the screen.

[0031]FIG. 1 depicts a system 10 according to the invention. The system10 is shown as a functional block diagram of a computer device of thetype that commonly includes a processor, a memory and a display.However, the system 10 may also be realized, in whole or in part as asoftware system comprising system code capable of executing on aprocessor to configure the processor as a system according to theinvention. The depicted system 10 includes a computer process 8, aplurality of source documents 11, a tool document file 30, a shapeprocessor 22 and a video display 26. The computer process 8 includes aplurality of document agents 12, a library of generic data objects 16,an internal representation file 14, a memory buffer Qr file 15, and aparser/renderer engine 18.

[0032] In the depicted embodiment, the display 26 can present the imagesof a plurality of different documents. Each of the representativeoutputs appearing on display 26 is termed a document, and each of thedepicted documents can be associated with one separate applicationprogram, such as Word, Netscape Navigator, Real Player, Adobe, Visio andother types of applications. It will be understood that the termdocument as used herein will encompass documents, streamed video, webpages, and any other form of data that can be processed and displayed bythe computer process 8.

[0033] The computer process 8 generates a single output display thatincludes within that display one or more of the documents. Thecollection of displayed documents represent the content generated by theapplication programs and this content is displayed within the programwindow generated by the computer process 8. The program window for thecomputer process 8 may also include a set of icons representative oftools provided with the graphical user interface and capable of allowinga user to control the operation, in this case the display, of thedocuments appearing in the program window.

[0034] For the illustrated embodiment, the display 26 presents contentrepresentative of different data types in a single, integrated display.This is in contrast to the conventional approach of having eachapplication program form its own display, which results in apresentation on the display device 26 that includes several programwindows, typically one for each application program. Additionally, eachdifferent type of program window would include a different set of userinterface tools for manipulating the content displayed in that window.Thus, the system depicted in FIG. 1 creates an integrated display thatcontains viewable images of documents of different types. This includesweb pages that would normally be viewed in a browser, word documentsthat would normally be viewed in a viewer or word processing document,PDF documents that would normally be viewed in a vector graphic reader,and streaming video that would normally be viewed in a video player.Thus, the depicted system 10 separates the content of these documentsfrom the underlying application program and presents them for display onthe screen 26.

[0035] To allow a user to manipulate the depicted documents, the system10 depicted in FIG. 1 provides a set of tools that can be used tonavigate through a collection of documents, whether it is a multi-pagetext document, the pages of a web site or a series of time varyingimages that make up a video presentation. To this end, as will beexplained in greater detail below, the system 10 creates documents thatare representative of tools and which may be displayed by the system 10just as system 10 would display any other type of document. Thus thesystem 10 of the invention has the advantage of providing a consistentuser interface, and only requiring knowledge of one set of tools fordisplaying and controlling the different documents.

[0036] As discussed above, each source document 11 is associated with adocument agent 12 that is capable of translating the incoming documentinto an internal representation of the content of that source document11. To identify the appropriate document agent 12 to process a sourcedocument 11, the system 10 of FIG. 1 includes an application dispatcher(not shown) that controls the interface between application programs andthe system 10. In one practice, an external application programminginterface (API) communicates with the application dispatcher whichpasses data, calls the appropriate document agent 12, or otherwisecarries out a request made by an application program. To select theappropriate document agent 12 for a particular source document 11, theapplication dispatcher advertises the source document 11 to all theloaded document agents 12. These document agents 12 then respond withinformation regarding their particular suitability for translating thecontent of the published source document 11. Once the document agents 12have responded, the application dispatcher selects a document agent 12and passes a pointer, such as a URI of the source document 11, to theselected document agent 12.

[0037] As shown in FIG. 1, the document agent 12 employs the library 16of standard object types to generate the internal representation 14,which describes the content of the source document 11 in terms of acollection of document objects as defined in the library 16, togetherwith parameters defining the properties of specific instances of thevarious document objects within the document. The document object typesemployed in the internal representation 14 will typically include: text,bitmap graphics and vector graphics which may or may not be animated andwhich may be two- or three-dimensional: video, audio and a variety oftypes of interactive objects such as buttons and icons. Vector graphicsdocument objects may be PostScript-like paths with specified fill andtransparency. Text document objects may declare a region of stylizedtext.

[0038] Once documents are translated into an internal representation ofdocument objects; these objects are passed to the parser/renderer 18.The parser/renderer 18 generates a context-specific representation or“view” of the documents represented by the internal representation 14.The required view may be of all the documents, a whole document or ofparts of one or some of the documents. The parser/renderer 18 receivesview control inputs which define the viewing context and any relatedtemporal parameters of the specific document view which is to begenerated. For example, the system 10 may be required to generate azoomed view of part of a document, and then to pan or scroll the zoomedview to display adjacent portions of the document. The view controlinputs are interpreted by the parser/renderer 18 to determine whichparts of the internal representation are required for a particular viewand how, when and for how long the view is to be displayed.

[0039] The context-specific representation/view is expressed in terms ofprimitive figures and parameters. Optionally, there may be a feedbackpath 42 between the parser/renderer 18 and the internal representation14, e.g. for the purpose of triggering an update of the content of theinternal representation 14, such as in the case where the sourcedocument 11 represented by the internal representation 14 comprises atime varying multi-frame animation.

[0040] Each source document 11 provides a digital representation of adocument, such as a text document, a spread sheet or some otherdocument. The document agent 12 creates an internal representation ofthat document. In one practice the created digital representationincludes information that describes the page layout of the document,including information about page size, margins and other page layoutinformation. The digital representation also includes information aboutthe content of the source document, such as the text, figures, and othercontent information that appears in the document. Processes fortranslating a known file structure into another structure are known inthe art, including systems that identify page structure and contentinformation. Any of the suitable techniques for performing thisoperation may be practiced without departing from the scope of theinvention.

[0041] The output from the parser/renderer 18 expresses the document interms of primitive figures. For each document object, the representationfrom the parser/renderer 18 defines the object at least in terms of aphysical, rectangular boundary box, the actual shape of the objectbounded by the boundary box, the data content of the object, and itstransparency. The shape processor 22 interprets the primitive object andconverts it into an output frame format appropriate to the target outputdevice 26; e.g. a dot-map for a printer, vector instruction set for aplotter, or bitmap for a display device. An output control input 44connects to the shape processor 22 and can deliver user interfacecontrol signals to generate an output suitable for a particular outputdevice 26. Thus, the parser/renderer 18 and the shape processor 22 canact as an engine that renders portions of the page layout and pagecontent on the display 26.

[0042] Additionally, FIG. 1 depicts a tool document file 30. The tooldocument file 30 may be a computer data file that stores informationrepresentative of an image, wherein that image may represent a tool suchas a magnifying glass, a cursor, a ruler, or any other type of tool. Forthe purpose of illustration, the system 10 depicted in FIG. 1 will nowbe described with reference to an example wherein the tool document file30 includes data that is representative of a graphical image of amagnifying glass. The magnifying glass image will be associated with afunction that allows a user to magnify the image of a document stored onthe display 26 by passing the magnifying glass over the respectiveimage. As will be described in greater detail hereafter, the magnifyingglass can include a central lens portion wherein portions of a documentthat fall under the lens of the magnifying glass appear to the user tobe magnified and therefore are presented in an enlarged format relativeto the rest of the underlying document. Although the below example willbe described primarily with reference to the magnifying glass tool, itwill be obvious to those of ordinary skill in the art that other typesof tools may be provided using the systems and methods described hereinand all such tools will be understood to fall within the scope of theinvention.

[0043] Turning to FIG. 2, the operation of the magnifying glass tool canbe seen. Specifically, FIG. 2 depicts the display 26 wherein the display26 presents a screen document 42 that comprises a plurality of subelements including the document 44, the thumbnail document 46, themagnifying glass document 48, and the lens document 50. The display 26presents the screen 42 as a single integrated document that contains subdocuments 44 through 50. The content provided for creating the screen 42can come from one or a plurality of source documents 11 whose contentappears as the document 44 and thumbnail document 46. The screendocument 42 also comprises content provided by the tool document file 30that, in this example, contains data according to an internalrepresentation data format wherein that data represents the image of themagnifying glass 48. Additionally, the tool document file 30 may containa portal object that creates a further document by processing theappropriate portion of the screen document 42 to present that content inan enlarged format to appear as the magnified document 50 appearingwithin the lens of the magnifying glass 48. Thus, the document appearingwithin the lens 50 is derived from the underlying document, andtherefore this derived document changes according to the context inwhich the magnifying glass tool 48 is employed. Accordingly, thespecific behavior for the tool can vary depending on the context of itsuse. For example, a magnifying glass tool may be associated with toolcode that processes differently the content of a content document havingmap data than a content document having text. For example, with a map,the magnifying glass tool may process the associated content document torender handles within the associated document structure that are taggedas only to be shown within a view created by a magnifying glass. Thusthe derived document presented within the magnifying glass tool 48 caninclude additional information, such as street names, tourist sites,public transportation locations, notations or other information. In thisoperation, the magnifying glass tool responds to the context of theapplication, which is the rendering of a view of a map. In otherapplications, where the magnifying glass tool is employed on text, thebehavior of the tool may result in changes to the color or style of thetext, or could result in the presentation of text editing tools and userinterface controls, such as control buttons, pull down menus, annotationinformation, text bubbles, or other types of information.

[0044] Accordingly, the screen document 42 is an integration andaggregation of information contained within a source document 11 and atool document file 30. An application program associated with the tooldocument file 30 can process the appropriate content to create theenlarged view 50. The magnifying tool 48 and the associated source codeare capable of identifying that portion of the screen document 42 thatis to be presented in an enlarged format to create the enlarged view 50.The tool code is further capable of processing the selected content tocreate the enlarged view 50 and clip that enlarged view within the lensarea of the magnifying glass 48 to achieve the display affect of anenlarged region of the screen 26. Thus, the tool document and the sourcedocument 11 are in the same internal representation, and thus can bemerged into the screen document 42, that can be rendered by theparser/renderer 18.

[0045] In one embodiment, the graphical tool 50 may be moved over thescreen by dragging with a cursor, or if a touch-sensitive screen ispresent, by dragging a stylus or some other pointer across the screen ofthe display. To process this movement, the display 26 may include ascreen monitoring process for monitoring the screen of the display 26 todetect movement of a cursor, stylus or some other pointer across theimages of the documents presented on the screen. Such screen monitoringprocesses are known in the art and any suitable process may be employed.The monitor process, therefore allows a user sense of tactile controlover the visual representation of the document 44. The movementsdetected by the screen monitor process may be passed to an interfaceprocess that processes the detected motion to detect a motionrepresentative of a known command. The interface process may be aseparate process or may be part of the screen monitor process, as iscommon in the art. As the interface module detects commands to move thetool 50, a navigation module can create input signals that direct theparser/render 18 to create a new display for presentation to the user,where in the display will show the tool 50 repositioned as the userwanted.

[0046] Accordingly, the system depicted in FIG. 1 is capable ofproviding a graphical user interface tool that may be integrated into ascreen display that represents a single document which contains aplurality of sub documents, some of which sub documents include thegraphical tools themselves. The power of this approach allows for thedevelopment of novel graphical user interface tools that allow a user tomanipulate and view a document on a display and to simulate tactilecontrol over the depicted documents. These systems and methods areparticularly well suited to use on hand held and mobile computingplatforms where traditional input tools are lacking. Additionalgraphical user interface tools that may be provided by the systems andmethods described herein include the bubble thumbnail graphical tooldepicted in FIG. 3. Specifically, FIG. 3 depicts the screen display 26that includes a screen document 52 which comprises a plurality of subdocuments including the document 44 and the thumbnail documents 60through 72. As shown in FIG. 3, the document 44 may be presented as alarge document employing most of the viewing area of the display 26. Inthis embodiment, the thumbnail documents 60 through 72 are arranged in avertical column within the screen document 52 at a position adjacent theleft side of the display 26. The thumbnail documents 60 through 72 varyin size with the largest thumbnail document 60 being centrallypositioned within the vertical array of thumbnail documents.

[0047] As further shown by FIG. 3 as documents in the vertical arrayincrease in distance from the center document 60, the documents decreasein size. The measure of distance from the center document may berepresentative of the distance in pages from the document 44, or may berepresentative of some other measure of distance or difference, such asthe amount of time that has passed since the document was last viewed,the difference in alphabetical order, or some other characteristic. Thusdocuments 62 and 68, which are adjacent the central document 60 aresomewhat smaller than the document 60. Further documents 64 and 70 whichare adjacent documents 62 and 68 respectively and further from document60 are smaller than documents 64 and 68 still. The decrease in size ofdocuments continues from documents 66 and 72, each of which is stillsmaller. The impression created by the array of thumbnail documents 60through 72 is employed to indicate that document 60, the largestdocument, is representative of the document 44 being displayed withinthe largest viewing area of the screen document 52. Documents 62 through72 get smaller in proportion to the “distance” from the current viewingpage 60. Accordingly, the vertical column of thumbnail document 60through 72 provide a navigation tool that a user can employ forselecting a document to appear within the large viewing area of thedisplay 26. Additionally, the user can select a document within thevertical array of thumbnails to choose a new document to appear withinthe viewing area. For example, in those applications where the screendisplay 26 is a touch sensitive screen display, the user may activate anew document to appear within the viewing area by touching therespective thumbnail document within the array of documents 60 through72. In those applications where the user is provided a keyboard, or amouse, the user may employ that particular input device for selectingwhich of the documents within the array of documents that the user wouldlike to appear within the viewing area. In an optional embodiment, theuser can scroll through the thumbnails to find the document of interest.Optionally, scrolling through the thumbnail documents can result in thedocument 44 changing with the scrolling of the document. Alternatively,the scrolling of the thumbnail documents can occur independently fromany changing of the document 44, with the document 44 only changing whena new thumbnail document is selected.

[0048] Accordingly, as the systems and processes described herein mayemploy thumbnail images for queuing a user during navigation to generatethumbnail images, the systems and processes described herein can includeany suitable thumbnail generator processes including those known in theart, including those thumbnail generations that generate live, oranimated thumbnails.

[0049]FIG. 4 depicts a further embodiment of the systems and methodsdescribed herein wherein the magnifying tool, earlier shown in FIG. 2 isassociated with tool code that results in information not earlierpresented in a document to appear within the lens area of the magnifyingglass object. More specifically, FIG. 4 depicts a display 26 thatincludes a screen document 42, which in this view appears as a map. FIG.4 further depicts the magnifying glass tool 48 that includes the lensarea 50. As shown in FIG. 4 the magnifying tool 48 is positioned over aportion of the map 42. As described above, the tool code associated withthe magnifying glass 48 is capable of presenting an enlarged view of therelevant portion of the screen document 42. As additionally shown inFIG. 4 the magnified portion 50 also includes additional information.For example, in the mapping application depicted in FIG. 4, the enlargedview 50 may include additional mapping information such as minor roads,locations of interest, or other information relevant to the contentbeing magnified. Additionally however, the magnifying glass may beassociated with tool code that changes that color of the information, ora portion of the information, within the viewing area 50, or presentsuser interface information such as control buttons, pull down menus,annotation information, text bubbles, or other types of information.Accordingly, the specific behavior for the tool can vary depending thecontext of its use. For example, as described above the magnifying glasstool may be associated with tool code that processes differently thecontent of a content document having map data than a content documenthaving text. Thus, the systems described herein provide contextsensitive tools and processes.

[0050]FIGS. 5 and 6 depict further embodiments of graphical userinterface tools that may be provided by the systems and methodsdescribed herein. In particular FIG. 5 depicts a screen 26 that includesa screen document 42 that comprises two documents 80 and 81, and a ruler82. The two documents 80 and 81 are meant to represent similar types ofdocuments, each being text documents printed on the same size paper,such as A4 paper. However, as the scale of the presentation of document81 is larger than the scale of presentation for document 80, FIG. 5depicts the two documents 80 and 81 as pages of text, wherein one pageis larger than the other. Thus, documents 80 and 81 are similardocuments that have been rendered with different scaling factors. Asdepicted in FIG. 5 the ruler 82 may be a floating semi-transparent rulerthat shows the scale of each document and which can adapt to the scaleof the underlying object. This is depicted by the scale of the ruler 82increasing in size as the length of the ruler travels from document 80to document 81. FIG. 5 portrays that the scale of the ruler 82 changesin proportion to the scale of the underlying documents. Thus, the ruler82 provides a context sensitive user interface tool that is capable ofadjusting the scale of the ruler in response to the presentation scaleof the content. Turning to FIG. 6, a further user interface tool, thefloating semi-transparent query mark 84 is depicted wherein the querymark 84 may display annotations for the underlying object according FIG.6 depicts that the display 26 includes a screen document 42 thecomprises a document 80 and a floating semi-transparent query tool 84.The query tool 84 when activated, either by dragging an image of thequery tool onto the document, or by selecting a query tool icon alreadypositioned over a document, will present text 88 that may includeinformation representative of an annotation of the underlying document80.

[0051]FIG. 7a depicts one method for presenting to a user the availableuser interface tool. In particular, FIG. 7a depicts a screen 26 thatincludes a tool button 90. The tool button 90 provides a graphicalrepresentation of the magnifying tool 48. To activate the magnifyingtool 48 the user may click, either by use of a mouse, keypad or touchscreen, and drag from the magnifier tool button an image of the tool 48.By clicking on the tool button 90 the system will process informationfrom the tool document file to create the image of the document 48 shownin FIG. 7a. FIG. 7b depicts that the user interface can allow in onepractice the user to push the magnifying tool 48 off the screen 26,optionally in any direction. By pushing the tool off the screen 26 theuser removes the tool 48 and restores the icon, or the tool button 90 tothe screen.

[0052]FIGS. 8a and 8 b depict a further tool of the type that may beemployed when viewing documents that include links, or other types ofpointers to other documents, or other content. Specifically FIG. 8depicts a tool wherein a document 100 includes links 102 to anotherdocument. For these documents, the systems and methods described hereinmay provide a slider control 104, as well as buttons, switches, or someother control. The depicted slider control 104, may enhance the userinterface view of the document 100 such that by sliding the control 104,the user can control the prominence of the link 102 within the document.Thus, the tool 104 allows the user to adjust the prominence of linkswithin a document such as the document 100 so that links may be moreeasily identified by the user. FIGS. 8a and 8 b further depict thatdocument 100 may include highlighted text such as the depictedhighlighted text 108. As with the link 102 the slider control 104 mayallow the highlighted text 108 to maintain its transparency while thetransparency of the remaining portions of document 100 vary as theslider control 104 is varied by the user. In operation, the slidercontrol 104 can allow the user to adjust the transparency, or alphafigure, of the objects that make up the document 100 other than theobjects that make up the link 102 or highlighted text 108. However,other techniques for fading or enhancing portions of a document, may bepracticed.

[0053] Turning to FIG. 9, a further graphical user interface tool ispresented wherein a document 100 includes links 102. Moreover the links102 may be associated with a floating thumbnail document 110. As shownin FIG. 9 a control 112 may be presented to the user. Upon activatingthe control 112, those links 102 within the document 100 may beassociated with a floating thumbnail representative of a page view ofthe page associated with the respective link 102. Additionally, FIG. 9depicts that in one option practice the display may further include acollegian of read ahead thumbnail documents 114. The thumbnail documents114 may be representative of those documents that are associated withlinks 102 within document 100, or that are associated with other pagesof the document 100 when that document 100 is a multipage document.

[0054]FIG. 10 depicts a further example of a graphical user interfacetool according to the invention. Specifically, FIG. 10 depicts a handheld computing device 120 having a set of characters 122 appearing onits display. As further shown in FIG. 10 a cursor window 124 appearsover one character within text display 122. In the depicted embodimentthe cursor window 124 provides a soft, semi-transparent text entry platefloating over the current text position. The plate may move with themoving text position and/or the plate may maintain its position as thetext itself scrolls to the left to accommodate movement of text underthe plate 124. As discussed above, the text plate cursor 124 may resultfrom a tool document file 30 processed by the system 10 of FIG. 1. Thetool document file may include an internal representation of the textplate 124 appears on the device 120. In one embodiment, the hand helddevice 120 includes a touch sensitive screen that allows a user toemploy a stylus for forming characters that will appear on the screenwithin the text entry plate 124. A design and development of suchsystems that allows such text entry are well known in the art and any ofthe suitable systems may be employed with the systems and methodsdescribed herein. In operation, a user may move a stylus across thescreen of the device 120 to form letters that will appear within thetext entry plate 124. This operation is depicted in FIG. 11 wherein aseries of text entry procedures 130 through 138 are depicted.

[0055] Particularly, FIG. 11 depicts text entry step 130 wherein thecursor plate 124 appears on the display of the device 120. A user maytrace a letter within space defined by the cursor 124, or in anotherarea and optionally, the tracings may appear within the area defined bythe text entry cursor 124. The tracings entered by the user may beprocessed by a character recognition system of the type known in the artto associate these markings with a character, such as the letter L inthis example. As shown in step 132 once character recognition has beencompleted the recognized character L may be presented on the display andthe cursor may move, or the text may scroll, but in either the cursor124 becomes available for the user to enter more text. Thus as shown instep 134 the user may enter text until a word is formed. In step 138,upon entry of a complete word the user may move the cursor 124 a spaceaway from the written word and begin again tracing characters that willappear within the text entry cursor 124. Thus, the depicted cursor 124provides a tool that allows in-line insertion of content into adocument, such as by inserting a section of text into an existing lineof text that occurs within the document. In other applications, toolscan be provided that edit images, such as by erasing content, changingcolors or performing other applications.

[0056] As described above, for those systems that include a touchsensitive display the systems and methods described herein may providefor allowing a user to use a stylus to trace markings on the display andthese marking may be interpreted by the system for allowing characterentry. Additionally, FIG. 12a through 12 b depicts a series ofrepresentative command strokes that a user may enter by moving stylusacross the touch sensitive screen. Each of the command strokes depictedin FIG. 12a through 12 g may be associated with a user interface commandthat the user may employ for manipulating and viewing documents. Forexample, 12 a depicts a stroke wherein the user forms a rounded checkmark that the system may be associated with a command that affirms anaction proposed by the system. Similarly, FIG. 12b depicts a stroke thatforms a plurality of peaks and troughs on the display and that mayassociated with a command to delete content from a display. FIG. 12cdepicts a circular clock wise stroke that may be associated with havinga document returned to a home page or to start over and FIG. 12d depictsa straight line diagonal upstroke that indicates a delete, clear, or nocommand. FIG. 12e depicts a box stroke running counter clock wise andindicating a paragraph select command and FIGS. 12f and 12 g depictstrokes that indicate a request by the user to move to the next orrespectively previous document. It will be understood by those ofordinary skill in the art that as the systems and methods describedherein include systems and methods that work with document of differenttypes such as Word documents, web pages, streaming media, and othertypes of content the meaning of the different strokes may vary accordingto the application. For example, the circular clock wise rotation of 12c may indicate for a document that is representative of web page arequest to return to a web page associated with the web page document.Alternatively, the use of the circular clock wise stroke of 12 c whenviewing streamed media content may indicate a request to start over,causing the streamed video to stop and restart from the beginning.Accordingly, it will be apparent to those with ordinary skill in the artthat the stroke command depicted in FIGS. 12a through 12 b may havedifferent meanings depending on their applications.

[0057]FIGS. 13a and FIG. 13b depict a command stroke that may beemployed by a user for clicking and dragging a document to cause pagemovement of that document within the viewing area. In the depictedembodiment, during a document drag operation document a velocitydetector process takes position readings periodically, such as everycenti-second. From these position readings a page velocity determinationmay be made. The page velocity determination may be employed forallowing the user interface to present a more natural way of movingdocuments through a viewing space. To this end, a process may employ thevelocity determination to direct the parser/render 18 to redraw thedocument in a series of pictures that will portray the document asmoving across the screen. For example, a user may drag a document at acertain speed and then release the stylus, mouse or other input devicefrom the document. Optionally, upon release the document may stopmoving. However, in an alternative practice the page may continue tomove in the established direction until the user indicates that thedocument is to stop moving such as clicking on the document. For multipage documents the velocity measure may be used for panning differentpages of the document across the screen at a rate determined by the pagevelocity set when the user drags one page of the document across thescreen. Optionally, the velocity may decrease by a constant page inertiauntil it reaches zero velocity and page scrolling ceases; during pagepanning further velocity detection can be used to increase (accumulate)the page velocity and hence movement against the page inertia enablingsmooth continuos movement of the page between rapid sequential dragoperations.

[0058] Additionally and optionally, other user interface processes maybe provided to enhance the user experience of having tactile controlover the document. For example, the user interface may include apage-flip detector for detecting a motion on the display 26 at alocation of the display associated with the upper right hand corner ofdocument 44 in FIG. 2. If the page-flip detector, or the screen monitor,detect a brushing motion across the surface of the document 44, thepage-flip detector can direct the parser/render 18 to “flip” the page,causing the next page, chapter, scene or other section to be displayed.Motions may be detected in either direction for page-flipping back andforth, and the page flip detector may be context sensitive, generating anew display suited to the application and type of content. Optionally,the interface process may include a page curl detector that can operatesimilar to the page-flip detector, except that a motion in the upperright corner of document 44 can cause the page curl detector to directthe parser/render 18 to redraw the screen 42 or document 44 so that thecorner of the document 44 is curled downward and a portion of theunderlying page is presented. Both the page-flip and page-curl detectorsmay be computer processes that can generate instructions to theparser/renderer 18 to achieve the desired effect. Additionally, apage-zoom detector (such as a double-click over the page area) can befollowed by an upward/downward movement to zoom in/out of the view. Thisfunction may be advantageously combined with the velocity detector toprovide an inertial zoom feature.

[0059] It shall be obvious to those of skill in the art that althoughFIG. 1 graphically depicts the user interface system 10 as functionalblock elements, these elements can be realized as computer programs orportions of computer programs that are capable of running on a dataprocessor platform to thereby configure the data processor as a systemaccording to the invention. Moreover, although FIG. 1 depicts the system10 as an integrated unit, it will be apparent to those or ordinary skillin the art that this is only one embodiment, and that the invention canbe embodied as a computer program distributed across multiple platforms.

[0060] As discussed above, the user interface systems described abovecan be realized as a software component operating on a data processingsystem, including hand-held computing platforms, as well as moreconventional computing platforms, such as a Unix workstation. In theseembodiments, the user interface systems can be implemented as a Clanguage computer program, or a computer program written in any highlevel language including C++, Fortran, Java or BASIC. Additionally, inan embodiment where the platform is primarily a microprocessor,microcontrollers or DSPs, the user interface systems can be realized asa computer program written in microcode or written in a high levellanguage and compiled down to microcode that can be executed on theplatform employed. The development of such systems is known to those ofskill in the art, and such techniques are set forth in the literature,including for example Digital Signal Processing Applications with theTMS320 Family, Volumes I, II, and III, Texas Instruments (1990).Additionally, general techniques for high level programming are known,and set forth in, for example, Stephen G. Kochan, Programming in C,Hayden Publishing (1983). It is noted that DSPs are particularly suitedfor implementing signal processing functions, including preprocessingfunctions such as image enhancement through adjustments in contrast,edge definition and brightness. Developing code for the DSP andmicrocontroller systems follows from principles well known in the art.

[0061] Additionally, it is to be understood that although FIG. 1graphically depicts the computer process 8 as comprising a plurality offunctional block elements, these elements can be realized as computerprograms or portions of computer programs that are capable of running onthe data processing platform to thereby configure the data processingplatform as a system according to the invention. Moreover, although FIG.1 depicts the system 10 as an integrated unit of a process 8 and adisplay device 26, it will be apparent to those of ordinary skill in theart that this is only one embodiment, and that the systems describedherein can be realized through other architectures and arrangements,including system architectures that separate the document processingfunctions and user interface functions of the process 8 from thedocument display operation performed by the display 26.

[0062] Those skilled in the art will know or be able to ascertain usingno more than routine experimentation, many equivalents to theembodiments and practices described herein. Moreover, the systems andprocesses of the invention have wide application and can be employed ina range of devices including hand-held computers, telephones, mobiledata terminals, set top boxes, an embedded processor, a notebookcomputer, a computer workstation, a printer, a copier, facsimile machineand other systems. Additionally, it will be understood by those of skillin the art, that the systems described herein may be practiced with anysuitable interface devices, including touch-sensitive screens and pads,mouse input devices, keyboards and keypads, joysticks, thumb wheeldevices, a mouse, a trackball, virtual reality input systems, voicecontrol systems, eye movement control systems, and any other suitabledevices. Thus, it will also be understood that the systems describedherein have many uses and provide advantages over the prior artincluding providing a set of interface processes and systems thatprovide sophisticated manipulation of different document types.

[0063] Accordingly, it will be understood that the invention is not tobe limited to the embodiments disclosed herein, but is to be understoodfrom the following claims, which are to be interpreted as broadly asallowed under the law.

1-38 (Canceled)
 39. A method for providing a user-interface, comprisingi. providing a content document file having an internal representationof a document, which describes the document as a collection of documentobjects and parameters defining properties of instances of the objectswithin the document; ii. providing a tool document file, representativeof a graphical tool that performs a user interface function, having aninternal representation expressed in the same object and parameter basedrepresentation; iii. providing tool code associated with the tooldocument file; iv. generating a screen document for display that is anaggregation of the content document file and the tool document file; v.parsing the aggregated internal representation of the screen document;and vi. rendering the screen document to create a single output displaythat integrates the content document with the graphical tool thatperforms the user interface function.
 40. A method according to claim 39wherein providing the content document file includes providing adocument file representative of a plurality of source documents.
 41. Amethod according to claim 40 wherein the plurality of source documentscomprise different data formats.
 42. A method according to claim 39,wherein the tool code comprises a script.
 43. A method according to anyone of claims 39 to 42, wherein the tool document file is selected fromthe group of a user interface control tool or window/desktop furniture.44. A method according to any one of claims 39 to 42, wherein the tooldocument file is representative of interactive objects selected from thegroup consisting of a button, an icon, a pull down menu, a switch, and aslider control.
 45. A method according to any one of claims 39 to 42,wherein the tool document file includes information representative of agraphical tool selected from the group consisting of a magnifying glass,a ruler, a text entry cursor, a thumbnail navigation control, and aquery tool.
 46. A method according to claim 39, wherein: i. the toolcode associated with the tool document file is capable of processing thecontent document file or the tool document file to create a deriveddocument which forms part of the screen document; ii. the processedinternal representation of the derived document presents the content ina manner that achieves a display effect associated with the tool; andiii. the display effect is portrayed in a rendered screen document. 47.A method according to claim 46, wherein the processed internalrepresentation of the derived document changes according to a contextualrelationship among the graphical tool, the content document file, and anapplication program in which the tool document file is used.
 48. Amethod according to claim 47, wherein the contextual relationship isselected from the group consisting of a relative position of thegraphical interface tool and the rendered content, a time at which thegraphical interface tool acts on the rendered content, and a state ofthe rendered content.
 49. A method according to claim 39, furthercomprising: i. providing a means to move the graphical tool to aselected position over the rendered document on the screen, and ii.directing the tool code to process a portion of the content documentfile associated with a selected position.
 50. A method according toclaim 39, wherein providing the tool code comprises providing the toolcode for creating a display effect by altering document objects andparameters describing an internal representation of a document.
 51. Amethod according to claim 50, wherein altering document objects andparameters comprises modifying the internal representation to add acontent to the screen document.
 52. A method according to claim 39,wherein rendering the screen document comprises generating a view of thescreen document expressed in terms of primitive figures and parameters.53. A method according to claim 52 wherein the primitive figures aredefined in terms of a bounding box, a shape, a transparency, and a datacontent of the figure.
 54. A method according to claims 39 or 52,wherein providing the tool code comprises providing the tool code thatprocesses the generated view of the screen document to create a displayeffect by altering the parameters of the primitive figures that make upthe view of the screen document.
 55. A method according to claim 54wherein altering the parameters of the primitive figures comprisesaltering parameters selected from the group consisting essentially of ascale, a transparency, and a color of selected figures within the screendocument.
 56. A method according to claim 54 wherein processing thegenerated view of the screen document comprises clipping selectedfigures within the view of the screen document to a clipping areaassociated with the tool document file.
 57. A method according to claim39 wherein rendering the screen document comprises receiving a viewcontrol input that defines a viewing context and related temporalparameters to generate a context-specific view of the screen document.58. A method according to claim 57 wherein the context-specific view isselected from the group consisting of all of the document objects withinthe screen document, a whole document object, parts of one or some ofthe document objects within the screen document.
 59. The methodaccording to claim 57 wherein the view control input is interpreted todetermine which parts of the internal representation of the screendocument are required for the context-specific view.
 60. A methodaccording to claim 57 wherein the view control input is interpreted todetermine how, when and for how long the view is to be displayed.
 61. Amethod according to claim 39 wherein i. the graphical tool is presentedon the display by means of a tool button that may be activated by auser, and ii. activation of the tool button by the user results inprocessing of the tool document file to create an image of the graphicaltool within the display.
 62. A method according to claim 61, wherein: i.the tool code associated with the tool document file is capable ofprocessing the content document file or the tool document file to createa derived document which forms part of the screen document, ii. theprocessed internal representation of the derived document presents thecontent in a manner that achieves a display effect associated with thetool, and iii. the display effect is portrayed in the rendered screendocument when the user activates the tool button.
 63. A method accordingto claim 39 wherein the objects of the internal representation of thecontent document file and the tool document file are selected from thegroup consisting essentially of a text object, a bitmap graphic object,and a vector graphic object.
 64. A method according to claim 63, whereinthe object is animated.
 65. A method according to claim 63, wherein theobject is not animated.
 66. A method according to claim 63, wherein theobject is two-dimensional.
 67. A method according to claim 63, whereinthe object is three-dimensional.
 68. A method according to claim 39,wherein the object is selected from the group consisting of a videoobject, an audio object, and an interactive object.
 69. A methodaccording to claim 39, wherein the object is selected from the groupconsisting of a button, an icon, a pull down menu, a switch, and aslider control.